Skip to content

HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…#2054

Merged
sodonnel merged 1 commit intoapache:branch-2.10from
brfrn169:HDFS-15386-branch-2.10
Jun 9, 2020
Merged

HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…#2054
sodonnel merged 1 commit intoapache:branch-2.10from
brfrn169:HDFS-15386-branch-2.10

Conversation

@brfrn169
Copy link
Member

@brfrn169 brfrn169 commented Jun 5, 2020

…ing multiple DN's data directories

@hadoop-yetus
Copy link

hadoop-yetus commented Jun 5, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-2.10 Compile Tests _
-1 ❌ mvninstall 0m 16s root in branch-2.10 failed.
-1 ❌ compile 0m 11s hadoop-hdfs in branch-2.10 failed.
-0 ⚠️ checkstyle 0m 10s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 12s hadoop-hdfs in branch-2.10 failed.
-1 ❌ javadoc 0m 12s hadoop-hdfs in branch-2.10 failed.
+0 🆗 spotbugs 0m 49s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 11s hadoop-hdfs in branch-2.10 failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ javac 0m 11s hadoop-hdfs in the patch failed.
-0 ⚠️ checkstyle 0m 10s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 11s hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ javadoc 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ findbugs 0m 11s hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 11s hadoop-hdfs in the patch failed.
+0 🆗 asflicense 0m 14s ASF License check generated no output?
6m 31s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/Dockerfile
GITHUB PR #2054
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux c155e3a4b1b0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision branch-2.10 / b411fce
Default Java Oracle Corporation-1.7.0_95-b00
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-mvninstall-root.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/testReport/
Max. process+thread count 31 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jun 5, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 22m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-2.10 Compile Tests _
-1 ❌ mvninstall 0m 15s root in branch-2.10 failed.
-1 ❌ compile 0m 12s hadoop-hdfs in branch-2.10 failed.
-0 ⚠️ checkstyle 0m 12s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 14s hadoop-hdfs in branch-2.10 failed.
-1 ❌ javadoc 0m 13s hadoop-hdfs in branch-2.10 failed.
+0 🆗 spotbugs 1m 0s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 12s hadoop-hdfs in branch-2.10 failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 13s hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 14s hadoop-hdfs in the patch failed.
-1 ❌ javac 0m 14s hadoop-hdfs in the patch failed.
-0 ⚠️ checkstyle 0m 11s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 14s hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ javadoc 0m 13s hadoop-hdfs in the patch failed.
-1 ❌ findbugs 0m 12s hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 15s hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
28m 45s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/Dockerfile
GITHUB PR #2054
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux aec30aeb67b6 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision branch-2.10 / b411fce
Default Java Oracle Corporation-1.7.0_95-b00
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-mvninstall-root.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/testReport/
Max. process+thread count 32 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jun 7, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-2.10 Compile Tests _
-1 ❌ mvninstall 0m 15s root in branch-2.10 failed.
-1 ❌ compile 0m 13s hadoop-hdfs in branch-2.10 failed.
-0 ⚠️ checkstyle 0m 10s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 11s hadoop-hdfs in branch-2.10 failed.
-1 ❌ javadoc 0m 13s hadoop-hdfs in branch-2.10 failed.
+0 🆗 spotbugs 0m 49s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 11s hadoop-hdfs in branch-2.10 failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ javac 0m 11s hadoop-hdfs in the patch failed.
-0 ⚠️ checkstyle 0m 11s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 12s hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ javadoc 0m 12s hadoop-hdfs in the patch failed.
-1 ❌ findbugs 0m 11s hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 11s hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
6m 58s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/Dockerfile
GITHUB PR #2054
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux e4789a3a9e79 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision branch-2.10 / b411fce
Default Java Oracle Corporation-1.7.0_95-b00
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-mvninstall-root.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/testReport/
Max. process+thread count 32 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@brfrn169
Copy link
Member Author

brfrn169 commented Jun 7, 2020

It looks like QA for branch-2.10 is broken? @sodonnel Can you please help me?

@hadoop-yetus
Copy link

hadoop-yetus commented Jun 9, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 11m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-2.10 Compile Tests _
-1 ❌ mvninstall 0m 16s root in branch-2.10 failed.
-1 ❌ compile 0m 12s hadoop-hdfs in branch-2.10 failed.
-0 ⚠️ checkstyle 0m 12s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 11s hadoop-hdfs in branch-2.10 failed.
-1 ❌ javadoc 0m 11s hadoop-hdfs in branch-2.10 failed.
+0 🆗 spotbugs 0m 52s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 12s hadoop-hdfs in branch-2.10 failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 11s hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 12s hadoop-hdfs in the patch failed.
-1 ❌ javac 0m 12s hadoop-hdfs in the patch failed.
-0 ⚠️ checkstyle 0m 11s The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 12s hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ javadoc 0m 13s hadoop-hdfs in the patch failed.
-1 ❌ findbugs 0m 12s hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 12s hadoop-hdfs in the patch failed.
+0 🆗 asflicense 0m 16s ASF License check generated no output?
16m 38s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/Dockerfile
GITHUB PR #2054
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 406910e1b801 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision branch-2.10 / 14ff617
Default Java Oracle Corporation-1.7.0_95-b00
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-mvninstall-root.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/testReport/
Max. process+thread count 31 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sodonnel
Copy link
Contributor

sodonnel commented Jun 9, 2020

The PR branch compiles fine locally and the new tests pass OK (all tests in TestFsDataSetImpl pass). We also know the CI run was clean on trunk.

Let me check with some people if we are good to just push this change in. We don't run CI on all the 3.x branchs after backporting from trunk, and the change here from 3.x is fairly minor.

@sodonnel
Copy link
Contributor

sodonnel commented Jun 9, 2020

This change looks good, and even though the CI isn't running, its a minor change compared to what was fine on trunk. I will go ahead and merge it.

@sodonnel sodonnel merged commit d1f4c8f into apache:branch-2.10 Jun 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@brfrn169 @hadoop-yetus @sodonnel